Assertion Message

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 370 of xUnit Test Patterns for the latest information.
How do we structure our test logic to know which assertion failed?

We include a descriptive string argument in each call to an Assertion Method.

Sketch Assertion Message embedded from Assertion Message.gif

We make tests self-checking (see Goals of Test Automation on page X) by including calls to Assertion Methods (page X) that specific the expected outcome. When a test fails, the Test Runner (page X) writes an entry to the test result log.

A well-crafted Assertion Message makes it very easy to determine which assertion failed and exactly what the symptoms were when it happened.

How It Works

Every Assertion Method takes an optional string parameter that is included in the failure log. When the condition being asserted isn't true, the Assertion Message is output to the Test Runner's log in addition to whatever output the assertion method normally generates.

When To Use It

There are two schools of thought on this subject. Test drivers who belong to the "single assertion per Test Method" school don't feel they need to include Assertion Messages because there is only one assertion to fail and therefore they always know exactly which assertion happened. They count on the Assertion Method to include the arguments (e.g. expected "x" but was "y") but they don't need to include a message

On the other hand, people who find themselves coding several or many assertion method calls in their tests should strongly consider including a message that at least distinguishes which assertion failed. This is especially important if the tests are frequently run using a Command Line Test Runner (see Test Runner).

Implementation Notes

Saying we need an message for each assertion method call is easy but what should we say in the message? It is useful to take a moment as we write each assertion and ask ourself what the person reading the failure log would hope to get out of it.

Variation: Assertion Identifying Message

When we have several assertions of the same type in the same Test Method (page X), we make it more difficult to determine exactly which one failed the test. By including some unique text in each Assertion Message, we can make it very easy to determine which assertion method call failed. A common practice is to use the name of the variable or attribute being asserted on as the message. This is very simple and requires very little thought. Another options would be to number the assertions; this would certainly be unique but may be less intuitive as it would require looking at the code to determine which assertion was failing.

Variation: Expectation Describing Message

When a test fails, we know what has actually happened. The big question is "what should have happened?" There are several ways of documenting this for the test reader. One would be using comments in the test code. A better solution is to include a description of the expectation in the Assertion Message. While this is done automatically for an Equality Assertion (see Assertion Method), we need to provide it ourselves for any Stated Outcome Assertions (see Assertion Method).

Variation: Argument Describing Message

Some types of Assertion Methods provide less helpful failure messages than others. Among the worst are Stated Outcome Assertions such as assertTrue(aBooleanExpression). When these fail, all we know is that the stated outcome did not occur. In these cases we can include the expression that was being evaluated (including the actual values) as part of the Assertion Message text so that the test maintainer can see from the failure log what was being evaluated and why it caused the test to fail.

Motivating Example

      assertTrue( a > b  );
      assertTrue( b > c );
Example assertTrueWithoutMessage embedded from java/com/xunitpatterns/misc/SampleAssertionUsage.java

This emits a failure message something like "Assertion Failed". From this, we cannot even tell which of the two Assertion Messages has failed. Not very useful, is it?

Refactoring Notes

Fixing this is a simple matter of adding one more parameter to each Assertion Method call. In this case, we want to communicate first of all that we are expecting 'a' to be greater than 'b' but it would also be useful to be able to see what the values of 'a' and 'b' actually were. We can add all this information into the Assertion Message through some judicious string concatenation.

Example: Expectation Describing Message

Here is the same test with the Argument Describing Message added:

      assertTrue( "Expected a > b but a was '" + a.toString() +
                        "' and b was '" + b.toString() + "'", 
                  a.gt(b) );
      assertTrue( "Expected b > c but b was '" + b.toString() +
                        "' and c was '" + c.toString + "'",
                  b > c );
Example assertTrueWithMessage embedded from java/com/xunitpatterns/misc/SampleAssertionUsage.java

This will now result in a useful failure message:

Assertion Failed. Expected a > b but a was '17' and b was '19'.

Of course,this would be even more meaningful if the variables had Intent Revealing Names[SBPP]!

Page generated at Wed Feb 09 16:39:45 +1100 2011